MetaGrad: Multiple Learning Rates in Online Learning

In online convex optimization it is well known that objective functions with curvature are much easier than arbitrary convex functions. Here we show that the regret can be significantly reduced even without curvature, in cases where there is a stable optimum to converge to. More precisely, the regret of existing methods is determined by the norms of the encountered gradients, and matching worst-case performance lower bounds tell us that this cannot be improved uniformly. Yet we argue that this is a rather pessimistic assessment of the complexity of the problem. We introduce a new parameter-free algorithm, called MetaGrad, for which the gradient norms in the regret are scaled down by the distance to the (unknown) optimum. So when the optimum is reasonably stable over time, making the algorithm converge, this new scaling leads to orders of magnitude smaller regret even when the gradients themselves do not vanish. MetaGrad does not require any manual tuning, but instead tunes a learning rate parameter automatically for the data. Unlike all previous methods with provable guarantees, its learning rates are not monotonically decreasing over time, but instead are based on a novel aggregation technique. We provide two versions of MetaGrad. The first maintains a full covariance matrix to guarantee the sharpest bounds for problems where we can afford update time quadratic in the dimension. The second version maintains only the diagonal. Its linear cost in the dimension makes it suitable for large-scale problems.